I think all these checks here should be done atomically without acquiring/releasing locks in between. So basically:

PyMutex_Lock(&interp->ceval.pending.mutex);
_PyEval_StopTheWorldAll(interp->runtime);
HEAD_LOCK(&_PyRuntime);
// ... do the checks and compare-exchange on finalization_guards
HEAD_UNLOCK(_PyRuntime);
_PyEval_StartTheWorldAll(interp->runtime);
PyMutex_Unlock(&interp->ceval.pending.mutex);

In other words, lift the HEAD_LOCK() out from runtime_has_subinterpreters and interp_has_threads

I stuck the pattern in ChatGPT and it agreed with @encukou that it's not UB, at least if struct PyThreadStateToken is kept opaque.

The typedef struct PyThreadStateToken PyThreadStateToken definition is fine with me, and will probably provide stricter type checking.

Is the current ordering here deliberate? I'm not sure if there's any issues, but the split of detach/attach across _PyEval_StartTheWorldAll and PyMutex_Unlock confuses me.

I would write it as:

        _PyEval_StartTheWorldAll(interp->runtime);
        PyMutex_Unlock(&interp->ceval.pending.mutex);

        // Temporarily let other threads execute
        _PyThreadState_Detach(tstate);
        _PyThreadState_Attach(tstate);

But it's also not clear to me whey we need _PyThreadState_Detach/Attach here. If other threads are running, won't we wait on them in either wait_for_thread_shutdown or when parking on the finalization_guards?

Yeah, I think that's an artifact of an older variation.

There is a bug here (found by Claude) where PyThreadState_Ensure/PyThreadState_Release with an already attached detaches on release when it should keep the active thread state.

In other words, we aren't properly distinguishing the two cases "already has a valid active thread state" and "doesn't have any attached thread state".

https://gist.github.com/colesbury/b6538312a898dd97fdd770b22fb3c338

We should use an affirmative tone. For example:

Use this function when an interpreter pointer or view cannot be supplied by the caller, for example, when a native threading library does not provide a void *arg parameter that could carry a :c:type:PyInterpreterGuard or :c:type:PyInterpreterView. In code that supports subinterpreters, prefer :c:func:PyInterpreterView_FromCurrent so the guard tracks the calling interpreter rather than the main one.

I'm not sure I understand the purpose of this in the tests. If the intent is to try to capture timing bugs, use something like pysleep. You can copy-paste those few lines of code if necessary.

I'd prefer we don't export this symbol since that complicates potential future bug fixes if we need to change or remove it.

I'll just remove it. My hope was that this would make some of the apparent thread safety issues reproduce more reliably, but it doesn't seem that it did.

I think this also has a bug. Let's say you have:

t1 = PyThreadState_Ensure(main_interp_guard); // counter 0 -> 1
Py_BEGIN_ALLOW_THREADS
t2 = PyThreadState_Ensure(main_interp_guard); // counter 1 -> 2
PyThreadState_Release(t2); // BUG! 2->1
Py_END_ALLOW_THREADS
PyThreadState_Release(t1);

The inner PyThreadState_Release should restore the thread state to "not attached", but it leaves it as attached due to the counter.

Hm, this looks tricky to solve. I guess we could turn PyThreadStateToken into an actual structure that holds the necessary state, but I'm not super excited about adding another heap allocation to PyThreadState_Ensure. I'll have to think about this one a little bit.

Ok, my solution was to add a new sentinel that says to detach the thread state upon calling PyThreadState_Release, regardless of the counter.